Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmdlib.sh: feed /dev/zero as qemu stdin #3793

Merged
merged 2 commits into from
May 8, 2024
Merged

Conversation

jlebon
Copy link
Member

@jlebon jlebon commented May 6, 2024

This is a follow-up to 79b15c8 ("cmdlib.sh: go back to using tail -F
for command output") which was subsequently reverted.

To summarize, it seems like in QEMU v8.2 (in f40), the guest sometimes
would hang when writing over virtio-serial if the device is hooked up to
the QEMU's stdio.

In testing, removing the <&- hack to close QEMU's stdin fixed it for
CoreOS CI but not Prow:

#3785 (comment)

I think I've narrowed it down to CoreOS CI (i.e. Jenkins) allocating a
tty and Prow not. When stdin is not a tty, QEMU would immediately gets
EOF if it tries to read anything. I'm not sure exactly what happens, but
I think the virtio-serial hang is linked to this (even though there's no
userspace code in the guest trying to read from the virtio-serial port).

Work around this by explicitly feeding /dev/zero to QEMU's stdin.

Copy link

openshift-ci bot commented May 6, 2024

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@jlebon
Copy link
Member Author

jlebon commented May 6, 2024

/test all

@jlebon
Copy link
Member Author

jlebon commented May 6, 2024

/retest

@jlebon jlebon force-pushed the pr/revert-tail branch from 1b601c5 to d4c3666 Compare May 7, 2024 18:29
@jlebon
Copy link
Member Author

jlebon commented May 7, 2024

/test all

@jlebon
Copy link
Member Author

jlebon commented May 7, 2024

/test rhcos

@jlebon jlebon force-pushed the pr/revert-tail branch from 5dad454 to c56d168 Compare May 7, 2024 19:45
@jlebon
Copy link
Member Author

jlebon commented May 7, 2024

/test all

jlebon added 2 commits May 7, 2024 16:12
This reverts commit 79b15c8.

I found a workaround for the issue that prompted that commit. See next
commit.
This is a follow-up to 79b15c8 ("cmdlib.sh: go back to using `tail -F`
for command output") which was subsequently reverted.

To summarize, it seems like in QEMU v8.2 (in f40), the guest sometimes
would hang when writing over virtio-serial if the device is hooked up to
the QEMU's stdio.

In testing, removing the `<&-` hack to close QEMU's stdin fixed it for
CoreOS CI but not Prow:

coreos#3785 (comment)

I think I've narrowed it down to CoreOS CI (i.e. Jenkins) allocating a
tty and Prow not. When stdin is not a tty, QEMU would immediately gets
EOF if it tries to read anything. I'm not sure exactly what happens, but
I think the virtio-serial hang is linked to this (even though there's no
userspace code in the guest trying to read from the virtio-serial port).

Work around this by explicitly feeding `/dev/zero` to QEMU's stdin.
@jlebon jlebon force-pushed the pr/revert-tail branch from c56d168 to c5adc99 Compare May 7, 2024 20:38
@jlebon
Copy link
Member Author

jlebon commented May 7, 2024

/test all

@jlebon jlebon changed the title DNM: Revert "cmdlib.sh: go back to using tail -F for command output" cmdlib.sh: feed /dev/zero as qemu stdin May 7, 2024
@jlebon jlebon marked this pull request as ready for review May 7, 2024 23:05
@jlebon
Copy link
Member Author

jlebon commented May 8, 2024

OK sweet! CI is happy. 🎉

Yes, I should try to get a minimal reproducer of the underlying issue and file a bug at some point, but... time. :(

Copy link
Member

@dustymabe dustymabe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jlebon jlebon merged commit bb60451 into coreos:main May 8, 2024
5 checks passed
@jlebon jlebon deleted the pr/revert-tail branch May 8, 2024 16:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants